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SERIAL  EXPERIMENTATION  FOR  THE  MANAGEMENT  AND 
EVALUATION  OF  COMMUNICATIONS  SYSTEMS* 


When  there  is  a decision  to  mount  a demonstration  of  new  communications 
systems  in  the  field,  controversy  about  the  role  of  evaluation  often  re- 
sults. Those  who  have  operational  responsibility  for  the  system  become 
its  advocates,  believing  it  to  be  sufficiently  robust  to  succeed.  They 
resent  research  that  would  divert  resources  from  the  central  purpose  of 
demonstrating  success.  In  contrast,  evaluators  anticipate  the  chance  of 
failure.  They  maintain  that  it  is  essential  to  construct  systematic  and 
rigorous  research  on  a project  so  that  future  projects  can  learn  from 
past  experiences. 

The  conflict  is  likely  to  intensify  when  the  demonstration  is  under- 
way and  something  starts  going  wrong.  Project  personnel  will  quickly  want 
to  make  changes,  trying  to  correct  the  difficulty.  Such  action  is  clearly 
detrimental  to  the  evaluator's  efforts  since  the  more  rigorous  their 
design,  the  more  irrelevant  the  data  usually  will  be  if  the  program  has 
been  continually  altered.  But  try,  as  an  evaluator,  to  persuade  a program 
manager  that  he  or  she  should  leave  a falling  field  program  unaltered  so 
that  you  can  rigorously  establish  the  reason  for  failure  in  a final  report. 

One  approach  to  this  problem  which  serves  the  purposes  of  both  program 
advocates  and  evaluators  is  to  use  a research  design  that  consists  of  a 
series  of  short  experiments  that  can  mutate  through  successive  generations. 
The  manager  is  committed  to  holding  each  discrete,  experimental  inter-  \\ 
vention  constant,  permitting  rigorous  evaluation.  The  evaluator  is  commit- 
ted to  providing  the  results  of  each  experiment  to  the  manager  before  the 
sequential  experiment  is  launched,  and  being  prepared  to  design  new  instru- 
ments— however  Imperfect — on  the  spot  to  evaluate  this  change.  The  manager 
receives  superior  planning  data  in  return  for  periodic  inaction  and  con- 
comitant anxiety. 


This  paper  is  for  presentation  to  the  NATO  Science  Symposium  on 
The  Evaluation  and  Planning  of  Interpersonal  Telecommunications  Systems 
held  at  the  University  of  Bergamo,  September  5-9,  1977.  It  will  be  in- 
cluded in  the  collection.  Evaluation  and  Planning  of  Teleaonmmiaations 
Systems  (London:  Plenum  Publishing  Company,  Ltd.,  Winter  1977-78.) 
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To  serve  these  purposes,  a field  project  should  have  four  elements: 
serial  experiments,  a process  measure,  a robust  outcome  measure,  and  a 
standard  of  comparison.  The  process  measure  is  essential  to  understanding 
how  the  system  actually  operates  so  that  the  manager  can  be  given  guidance 
on  how  to  direct  change.  The  outcome  measure  should  be  tied  to  a broad 
goal  that  will  remain  relevant  despite  a variety  of  system  changes  so  that 
one  can  determine  the  relative  value  of  each  system  generation.  The 
standard  of  comparison  is  essential  to  a discussion  of  the  value  of  repli- 
cating the  system  and  the  generalizability  of  the  results. 

Having  these  factors  in  a project  design  has  almost  no  disadvantages, 
and  provides  many  opportunities  to  learn  from  demonstrations  even  when  the 
starting  assumptions  are  found  to  be  in  error.  There  are,  of  course,  many 
times  when  the  single,  grand  experiment  is  the  only  feasible  route,  but 
it  seems  likely  that  many  opportunities  to  establish  serial  experiments 
are  missed.  Here  we  report  on  the  results  of  using  this  logic  in  the 
Spartanburg  two-way  cable  project. 

The  Spartanburg  Interactive  Cable  Education  Program 

Since  February  1975,  a two-way  cable  project  has  been  underway  in 
Spartanburg,  South  Carolina,  that  compares  the  progre  s of  students  who  use 
simple  home  terminals  to  participate  interactively  in  televised  instruction 
with  the  progress  of  students  who  receive  conventional  classroom  instruction. 
This  effort  is  part  of  a program  of  research  supported  by  the  Research 
Applied  to  National  Needs  (RANN)  program  of  the  National  Science  Foundation 
on  the  costs  and  benefits  of  two-way  cable  television  systems,  systems  that 
can  be  used  both  to  send  and  receive  signals  from  a home,  agency,  or  busi- 
ness. Research  described  here  is  one  of  several  education  and  training 
programs  conducted  in  Spartanburg  designed  to  test  the  value  of  alternative 
forms  of  return  communications  on  a cable  system. 

Interactive  cable  television  offers  a technological  opportunity  for 
education  that  lies  somewhere  between  conventional  educational  television 
(ETV)  and  computer-assisted  instruction  (CAI) . Like  ETV,  students  watch 

^Segmented  time  series  data  can  often  be  used  in  lieu  of  serial  ex- 
periments, providing  many  of  the  same  advantages. 
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teachers  on  a television  set  and  the  programs  are  designed  to  make  heavy 
use  of  visual  and  workbook  materials.  Like  CAI,  the  students  have  data 
terminals  that  they  use  to  record  answers  to  the  teachers'  questions  and 
assigned  exercises.  With  interactive  cable,  however,  the  students'  answers 
are  immediately  corrected  and  reinforced  by  a teacher  rather  than  by  a 
computer.  This  not  only  benefits  the  students  by  providing  human  as 
opposed  to  machine  feedback,  but  it  also  enables  the  teacher  on  cable 
television  to  have  continuing  information  about  the  progress  of  each 
student  readily  available  as  the  class  proceeds. 

In  this  experiment,  adult  education  at  the  high  school  level  was  chosen 
as  the  substantive  content  to  test  the  value  of  two-way  cable  to  the  home. 
Using  the  two-way  capacity,  Spartanburg  Technical  College  has  offered  a 
series  of  home-based  adult  education  classes  using  an  interactive  cable 
television  system  in  English  grammer,  reading,  and  mathematics,  the  three 
subjects  necessary  to  prepare  students  for  the  General  Educational  Develop- 
ment (GED)  examination  for  a high  school  equivalency  degree.  The  text  mater- 
ial has  been  the  Cambridge  GED  series  and  the  content  of  the  course  is 
similar  to  that  of  many  GED  programs  around  the  country.  In  this  demon- 
stration project,  the  progress  of  students  in  the  conventional  classroom 
setting  is  compared  with  that  of  students  in  the  "electronic  classroom" 
who  view  classes  from  their  homes.  Each  of  these  students  has  available  an  8- 
button  hand-held  terminal  to  respond  to  questions  posed  by  the  television 
teacher  during  a class  period. 

Students  in  the  electronic  classes  can  use  their  terminals  for  three 
distinct  kinds  of  student-teacher  interactions.  The  most  formal  inter- 
action is  the  quiz  or  question  period.  In  this  mode,  the  teacher  asks 
multiple  choice  questions  from  a workbook  and  the  students  punch  in 
what  they  believe  to  be  the  correct  answers.  After  a brief  pause,  the 
teacher  enters  the  correct  answer  through  the  terminal  and  calls  up  a 
display  on  a CRT  mounted  in  a lectern.  She  can  easily  read  a list  of  the 
student  names,  the  answers  they  punch,  and  the  aggregate  number  of  right 
and  wrong  answers.  These  results  are  also  recorded  in  the  computer  memory 
so  that  at  the  end  of  the  class  the  teacher  has  a hard  copy  of  all  student 
responses  as  well  as  various  summary  statistics  that  aid  in  an  assessment 
of  individual  student  progress. 
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In  addition  to  questions  posed  in  the  question  mode,  the  teacher  can 
initiate  many  informal  questions,  such  as  questions  related  to  procedures 
and  student  understanding  of  formal  course  content.  Thus  the  students  can 
be  asked  to  indicate  whether  they  found  an  exercise  to  be  too  difficult, 
or  whether  they  have  completed  an  in-class  problem  that  has  been  assigned. 

A teacher  wishing  to  receive  this  kind  of  information  from  students  can 
switch  the  system  to  a second.  Informal  mode.  The  switch  commands  the 
system  logic  to  display  student  responses  for  the  teacher  but  not  to 
record  these  responses  in  the  diagnostics  being  compiled  in  the  computer 
memory. 

The  system  also  has  been  designed  to  allow  for  student  as  well  as 
teacher-initiated  communications.  When  the  teacher  is  not  asking  questions, 
she  can  put  the  system  into  a third  mode.  Then  when  a student  hits  a button 
his  or  her  name  and  an  alphanumeric  message  appear  on  the  teacher's  CRT 
instead  of  a number  for  a multiple-choice  response.  In  the  current  system, 
the  student  can  send  seven  messages.  These  are  "I  understand"  (indicating 
the  student  is  ready  to  move  on),  "slow  down,"  "give  an  example,"  "ask  a 
question"  (so  I can  see  if  I understand),  "visuals  are  unclear,"  "I  don't 
understand"  (so  please  repeat  and  review) , and  "call  me"  (on  the  telephone) . 
Each  of  these  messages  is  printed  on  the  student  terminals  by  the  appropriate 
button.  Even  though  these  messages  are  not  a total  substitution  for  the 
rich  array  of  both  verbal  and  nonverbal  cues  and  messages  that  a student 
sends  to  a teacher  in  a conventional  classroom,  they  do  provide  students 
with  a means  to  communicate  with  the  instructor  with  respect  to  how  the 
material  is  being  received.  Like  the  other  interactive  capabilities,  this 
capacity  for  student  initiated  signals  was  built  into  the  system  to 
make  the  experience  in  the  electronic  classroom  as  similar  as  possible  to 
that  in  the  regular  classroom  setting. 

Verbal  communication  was  also  possible  in  later  classes.  The  tele- 
phone was  not  used  in  the  first  class,  but  it  was  available  in  subsequent 
course  offerings.  A student  could  call  the  teacher — and  be  heard  over 
the  system  by  the  rest  of  class  should  the  teacher  choose  to  lead  a general 
discussion.  Telephone  use  was  occasional  and  supplemental  in  nature. 
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Standard  of  Comparison 

The  demonstration  has  been  structured  so  that  the  program  can  be 
evaluated  relative  to  conventional  classroom  education.  Each  day,  two 
teachers  Instruct  the  class,  and  their  schedules  are  rotated  so  that  both 
classes  receive  equivalent  instruction.  On  a typical  day,  the  math  teacher, 
for  example.  Instructs  her  class  over  cable  from  8:30  to  10:00  a.m.  and 
then  drives  to  Spartanburg  TEC  where  she  teaches  a regular  class  from 
10:30  until  noon.  The  reading  teacher  has  the  reverse  schedule,  instructing 
the  conventional  class  at  TEC  until  10:00  a.m.,  and  starting  the  same 
reading  lesson  over  cable  at  10:30  a.m.  The  curriculum  is  highly  structured 
and  is  geared  toward  preparing  students  for  the  type  of  question  that 
appears  on  the  GED  exam.  A large  portion  of  each  class  is  devoted  to  stu- 
dents working  on  and  discussing  the  answers  to  workbook  problems  and  exer- 
cises that  are  similar  to  those  on  the  State  GED  examinations.  Both  classes 
were  offered  4 days  a week  for  15  weeks,  a total  of  180  hours  of  instruction. 
In  sum,  students  in  both  cable  and  traditional  classes  receive  the  same 
amount  of  instruction  in  the  same  subjects  from  the  same  teachers  on  the 
same  days.  This  arrangement  enables  us  to  compare  the  effectiveness  of 
interactive  cable  education  in  the  home  with  conventional  education  in 
the  classroom  while  holding  many  factors  constant. 

Outcome  Measure 

The  basic  evaluation  measure  for  this  study  was  the  Adult  Basic 
Learning  Examination  (ABLE),  a standardized  test  of  educational  achieve- 
ment. The  measure  stood  out  because  it  had  acceptable  reliability,  and 
came  in  two  versions  for  a pretest-posttest  evaluation.  Moreover,  ABLE 
was  the  only  technically  sound  examination  that  had  been  normed  on  an 
adult  population.  This  would  enable  us  to  compare  the  scores  of  the 
Spartanburg  cohorts  entering  the  program  with  those  of  a national 
sample  of  adults  with  similar  educational  backgrounds.  This  was  important 
because  it  helped  us  establish  the  extent  to  which  the  skill  levels  of 
students  in  our  experiment  were  similar  to  the  skills  of  this  type  of 
student  in  other  parts  of  the  nation.  Being  able  to  show  that  the 
Spartanburg  students  are  not  atypical  permits  us  to  generalize  the 
results  of  our  study  to  other  adult  populations. 
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Need  for  Process  Information 

But  if  one  Is  to  provide  the  manager  some  guidance  on  the  experi- 
ment, it  is  also  necessary  to  provide  insight  into  the  ongoing  communications 
process.  We  therefore  felt  it  was  necessary  and  important  to  supplement 
the  outcome  data  with  process  information  that  focused  on  the  instructional 
dynamics  of  the  cable  and  conventional  classrooms.  This  process  informa- 
tion could  serve  at  least  three  different  roles  in  the  demonstration. 

First,  the  process  information  would  help  us  precisely  describe  the 
nature  of  the  cable  "classroom"  and  enable  us  to  understand  how  the  two- 
way  technology  affects  the  pedagogical  process.  By  comparing  systematic 
classroom  observation  data  from  both  the  conventional  and  cable  settings, 
we  could  establish  how  the  move  to  cable  had  altered  such  aspects  of 
instruction  as  organization  of  time,  the  nature  of  class  activities,  and 
patterns  of  classroom  interaction. 

Second,  by  providing  a tool  that  would  help  us  adapt  the  organization 
of  content  and  teaching  styles  to  the  potential  of  the  technology,  the 
process  information  would  be  useful  in  project  management.  We  did  not 
assume  that  the  instructional  d)niamlcs  we  would  observe  were  necessarily 
those  that  had  to  be.  In  some  cases  the  data  we  collected  might  suggest 
that  teachers  were  not  taking  full  advantage  of  the  technology.  Our 
observations  would  enable  us  to  offer  inservice  training  that  was  targeted 
at  particular  problem  areas.  However,  if  such  training  failed  to  result 
in  a realization  of  the  potential  of  the  technology,  we  would  have  to 
entertain  the  notion  that  teachers  are  not  able  to  satisfactorily  transfer 
their  teaching  styles  to  the  two-way  cable  setting.  Such  a finding  would 
have  important  implications  for  our  assessmen  ' the  educational  potential 
of  the  electronic  classroom  concept. 

Third,  the  data  from  the  observations  could  be  used  to  help  us  under- 
stand and  explain  differences  in  group  outcomes  if  they  occurred.  In  the 
event  that  student  achievement  in  two-way  cable  class  was  poor,  we  needed 
to  know  enough  about  the  dynamics  of  each  teacher's  style,  level  of  inter- 
action, and  use  of  class  time  to  identify  possible  remedies.  Alternatively, 
if  the  cable  class  did  well  relative  to  the  conventional  group,  it  would 
be  beneficial  to  document  the  teaching  style  that  had  contributed  to  this 


outcome. 
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The  Observation  Instrument.  These  three  uses  of  process  information 
led  to  our  decision  to  develop  a classroom  observation  instrument  that  would 
permit  us  to  compare  the  nature  of  the  cable  and  conventional  classes. 

The  underlying  approach  of  the  instrument  was  to  focus  on  those  areas 
where  one  might  expect  to  find  Important  differences  between  the  cable 
and  conventional  classes  and  that  would  either  indicate  that  the  teachers 
were  not  making  full  use  of  the  interactive  dimension  of  the  system  or 
that  the  conventional  and  cable  classrooms  were  proceeding  quite  differently 
from  one  another. 

The  Instrument  was  designed  to  focus  on  two  distinct  areas  of  class- 
room pedagogy,  the  observation  of  the  distribution  of  classroom  activities 
(the  activity  record)  and  observations  of  the  frequency  of  classroom 
interactions  (the  interaction  record).  Events  falling  in  either  of  the 
two  areas  are  coded  continuously  in  order  to  present  a moving  record 
of  the  instructional  dynamics  in  the  respective  classes. 

The  activity  record  is  a system  of  categories  used  to  code  the  number 
of  minutes  devoted  to  different  activities.  The  classroom  activity  variables 
(Figure  lA)  are  simply  descriptors  of  the  types  of  activities  the  students 
and  teachers  are  engaged  in  at  any  given  time.  The  coder  is  responsible 
for  tracking  both  the  order  in  which  those  activities  take  place  and  the 
amount  of  time  devoted  to  each  activity.  These  variables  were  intentionally 
defined  so  they  would  be  grouped  to  distinguish  along  three  different  dimen- 
sions that  we  believed  to  be  of  potential  importance  in  comparing  the 
instruction  available  to  students  in  conventional  and  electronic  settings. 

The  first  dimension  contrasts  activities  that  involve  subject  matter 
development  with  activities  not  related  to  the  instructional  focus  of  the 
course.  For  Instance,  in  a math  class  a period  of  time  may  be  characterized 
by  the  teacher  lecturing  on  how  to  solve  quadratic  equations  or  by  students 
working  at  their  desks  on  word  problems.  Both  of  these  activities  Involve 
the  "substance”  of  the  course — the  teaching  of  mathematics  to  students.  In 
contrast,  time  spent  on  such  activities  as  equipment  adjustment  or  discussion 
of  how  the  final  grades  will  be  determined  (l.e.,  procedural  information)  is 
not  considered  relevant  to  the  substantive  purpose  of  the  class. 

A second  way  of  comparing  activities  is  by  the  extent  to  which  they 
reflect  an  Interaction  between  the  teacher  and  students.  Several  of  the 
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activity  variables  have  been  included  that  indicate  a high  degree  of  student- 
teacher  interaction.  For  instance,  drill  and  substantive  drill  are  character- 
ized by  the  teacher  repeatedly  asking  questions  of  students.  Likewise,  for 
classroom  substantive  discussion  to  be  coded,  there  must  be  evidence  of 
student's  questions  and  comments  being  frequently  interspersed  during  a 
teacher's  presentation.  In  contrast,  other  activities  such  as  individual 
work  and  teacher  substantive  presentation  are  only  coded  when  the  activity 
in  the  classroom  is  characterized  by  little  or  no  interaction  between  the 
teacher  and  the  students. 

The  third  dimension  for  grouping  the  activity  variables  is  concerned 
with  whether  or  not  the  activity  represented  by  the  variables  can  appear 
in  both  the  cable  and  conventional  settings.  Some  of  the  activities  such 
as  drill,  individual  work  period  or  teacher  substantive  presentation  can 
be  coded  in  either  the  cable  or  conventional  classroom.  Others  such  as 
classroom  substantive  discussion  and  student  dominated  activity  can  only 
take  place  in  the  traditional  classroom.  Similarly,  equipment  adjustment 
only  makes  sense  in  the  context  of  the  cable  classroom.  By  including 
these  variables,  we  can  determine  the  extent  to  which  teachers  are 
prevented  from  translating  aspects  of  their  traditional  teaching  styles 
to  the  cable  setting  by  virtue  of  the  fact  that  the  cable  setting  will 
not  accommodate  some  activities.  Similarly,  we  can  determine  how  great 
a role  the  unpredictable  demands  of  the  technology  play  in  contributing  to 
the  differing  experience  of  students  in  the  cable  and  conventional  settings. 

The  second  part  of  the  classroom  observation  Instrument  is  the  inter- 
action record.  It  deals  with  the  nature  and  number  of  teacher  and  student 
initiated  questions.  The  observer  uses  this  portion  of  the  instrument  to 
keep  an  ongoing  record  of  the  types  and  frequencies  of  questions  asked  in 
the  classroom.  The  observer  also  notes  whether  the  questions  are  directed 
at  a specific  individual  or  at  the  class  as  a whole.  Figure  IB  presents 
criteria  for  coding  different  interaction  variables. 

The  interaction  variables  in  this  second  category  system  have  been 
defined  with  distinct  contrasts  in  mind  that  are  similar  to  those  previously 
described  for  the  activity  variables.  On  one  level,  the  variables  distin- 
guish between  those  questions  that  relate  to  the  subject  matter  content 
being  studied  (l.e.,  the  open,  closed,  and  understand  questions)  from  those 
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that  deal  with  class  procedures  (i.e.,  the  routine  questions).  On  another 
level,  the  variables  can  be  used  to  distinguish  those  interactions  that  can 
take  place  in  either  the  cable  or  conventional  classroom  (e.g.,  the  asking 
of  closed-ended  question  or  questions  of  understanding)  from  those  that  can 
only  occur  in  the  conventional  class  (e.g.,  the  asking  of  an  open-ended 
question,  or  the  occurrence  of  student-student  interaction  or  teacher-stu- 
dent interaction  during  an  individual  work  period).  In  addition,  several 
variable  categories  require  that  the  coder  keep  track  of  those  instances 
where  the  teacher  asks  students  to  explain  some  aspect  of  their  response. 

Such  an  explanation  is  clearly  not  possible  over  the  cable  system. 

Measuring  the  Impact  on  the  Teaching  Process 

The  results  of  the  first  classroom  observation  records  showed  clearly 
that  the  pedagogical  process  over  interactive  cable  differs  dramatically 
from  that  of  the  traditional  classroom.  Based  on  a sample  of  over  100  ten 
minute  observation  periods,  both  of  the  cable  instruction  and  the  convetn- 
tional  classroom,  major  differences  were  evident  (Table  1).  Note  that 
teachers  devoted  18.2  percent  of  the  conventional  classroom  time  to  sub- 
stantive classroom  discussion.  Of  course,  there  was  no  opportunity  for 
substantive  classroom  discussion  over  cable,  and  thus  a central  question 
was  how  the  teachers  sought  to  compensate  for  that  loss  of  time.  It  was 
not  by  extending  lectures,  for  the  amount  of  time  devoted  to  substantive 
presentation  was  relatively  constant,  12.0  percent  of  classroom  versus 
13.1  percent  over  cable.  The  difference  instead  appeared  in  the  amount  of 
time  devoted  to  individual  student  work.  While  the  teachers  had  the  stu- 
dents working  individually  on  problems  at  their  desks  29.7  percent  of  the 
time  in  the  classroom,  the  proportion  rose  to  an  alarming  59.8  percent 
with  the  cable  classes. 

The  interaction  records  also  showed  the  Impact  of  the  technology.  The 
absolute  frequency  of  interactions  in  the  cable  classroom  dropped  off  to  one- 
third  of  the  total  interactions  in  the  traditional  setting.  Norming  the 
data  to  estimate  the  average  number  of  Interactions  in  a 90-mlnute  class 
period,  we  found  that  the  teacher  addressed  the  classroom  students  with 
questions  about  138  times  per  class,  compared  to  45  times  per  cable  class. 

The  number  of  routine  procedural  questions  was  not  very  different,  but  there 
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Table  1 

DISTRIBUTION  OF  ACTIVITIES  AND  INTERACTIONS  BY  CONDITION 


Distribution  of  Activity  Time 

Classroom 

Cable 

Teacher  Substantive  Presentation 

12.0% 

13.1% 

Classroom  Substantive  Discussion 

18.2 

0.0 

Individual  Work 

29.7 

59.8 

Drill 

0.0 

0.6 

Substantive  Drill 

24.7 

17.5 

Procedural  Information 

9.3 

7.2 

Student-Dominated  Activity 

0.0 

0.0 

Exam 

0.0 

0.0 

Equipment  Adjustment 

0.0 

1.0 

Teacher  Works  with  Subgroup 

1.6  ] 

0.0 

Nondesignated  Activity 

4.5 

0.8 

Total 

100.0% 

100.0% 

Frequency  of  Interactions  per  Class 

Classroom 

Cable 

Closed-ended  Questions 

14.0 

17.4 

Open-ended  Questions 

35.2 

0.0 

Questions  about  Understanding 

47.6 

9.8 

Procedural  Questions 

13.8 

12.7 

Rhetorical  Questions 

12.9 

5.4 

Student  Dominated  Activity 

14.8 

0.0 

Total  Frequencies 

138.3 

45.3 

Total  Minutes  of  Observation 

519 

494 
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was  a marked  decline  in  the  number  of  questions  about  student  comprehension 
of  the  material,  as  well  as  a decline  in  rhetorical  questions.  While  the 
cable  class  was  asked  slightly  more  structured,  close-ended  questions  of 
the  type  the  data  terminals  facilitate,  the  rise  failed  to  compensate 
for  the  loss  of  the  47. 6 open-ended  questions  the  teachers  asked  the 
average  conventional  class.  And  there  were  no  student-initiated  ques- 
tions or  student-to-student  Interactions  over  the  cable  in  the  first  round.' 

The  shock  was  that  only  ten  students  had  enrolled.  In  the  City  of 
Spartanburg,  62  percent  of  the  adults  have  not  completed  their  high  school 
education.  The  evidence  collected  in  the  planning  stage  of  the  project 
suggested  that  many  adults  dropped  out  because  of  lack  of  transportation, 
or  the  need  to  take  care  of  children  at  home.  Cable  had  been  assumed 
to  circumvent  these  barriers.  The  first  hypothesis  was  that  a program 
may  need  time  to  win  acceptance  and  we  assumed  that  enrollment  would 
grow. 

The  first  class  size  comprised  our  ability  to  reach  conclusions 
on  the  test  data,  but  the  results  were  encouraging.  The  two  classes 
had  made  roughly  equivalent  gains  despite  the  differences  in  the 
teaching  process.  Of  the  ten  students  that  started  the  cable  class,  all 
were  sufficiently  motivated  to  complete  the  180  hours  of  instruction 
despite  the  lowered  levels  of  Interaction  and  the  absence  of  personal 
contact.  Moreover,  as  measured  by  the  Adult  Basic  Learning  Examination, 
their  progress  was  comparable  to  that  achieved  by  the  13  of  25  students 
that  completed  the  full  series  of  conventional  classes.  There  was,  of 
course,  the  usual  problem  of  interpretation  because  the  students  were 
unavoidably  aware  of  their  uniqueness  as  participants  in  the  first 
class  of  its  kind,  and  special  attention  effects  may  have  also  played 
a motivating  role. 

In  this  case,  the  decision  was  made  to  keep  the  second  experiment 
relatively  constant.  As  best  we  could  tell  from  the  ABLE,  the  basic  con- 
cept was  effective.  Now  faced  with  the  danger  that  our  student  samples 


Student  use  of  data  terminals  to  signal  a request  for  a change  in 
the  pace  of  the  lecture  or  for  a review  were  not  coded  since,  among  other 
problems,  it  is  not  clear  if  that  is  equivalent  to  verbal  or  non-verbal 
signals  in  the  average  discussion.  In  the  second  round  when  a telephone 
was  available  the  students  could  initiate  verbal  questions. 
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would  be  much  too  small,  we  decided  to  repeat  the  program.  Since  the 
teachers,  curriculum,  technology,  and  goals  were  the  same,  we  planned  to 
pool  the  student  data  if  the  student  numbers  continued  to  be  small.  Using 
the  classroom  observation  data,  we  would  try  to  improve  teacher  performance 
using  the  technology,  even  though  this  action  would  mean  that  there  would 
be  some  differences  in  the  class  dynamics  between  the  first  and  subsequent 
experiments. 

Adjusting  Teacher  Style  in  a New  Media  Environment 

When  the  teachers  were  shown  the  results  of  the  first  round  of  obser- 
vation data,  they  were  surprised  by  the  magnitude  of  the  differences  in  the 
pedagogical  process  in  the  cable  and  traditional  classrooms.  In  particu- 
lar, they  agreed  that  far  too  much  of  the  cable  class  time  had  been  spent 
in  individual  work  while  too  little  time  had  been  devoted  to  substantive 
drill.  As  they  reflected  on  the  reasons  for  the  disparity  in  their  class- 
room behavior,  it  became  evident  that  the  absence  of  nonverbal  cues  from 
students  in  the  cable  condition  was  leading  them  to  prolong  the  work 
periods.  In  both  the  classroom  and  cable  class,  students  could  indicate 
when  they  had  completed  their  assignment,  but  in  the  classroom  the  stu- 
dents could  then  exert  substantial,  largely  non-verbal  pressure  on  the 
teacher  to  move  on.  Thus  the  teacher  was  waiting  longer  for  the  slower 
students  in  the  cable  environment.  In  addition,  the  teachers  agreed  to 
put  more  emphasis  on  drill.  The  potential  of  the  technology  to  Involve 
all  of  a class  in  answering  questions  was  more  evident,  and  the  teachers 
felt  that  this  strength  of  the  interactive  system  should  be  used  more 
fully. 

In  the  fall  of  1976,  the  GED  class  was  offered  again  along  with  other 
interactive  cable  programs.  The  class  was  again  small,  and  th*  only  technical 
change  was  that  the  cable  students  were  permitted  to  use  their  home  tele- 
phones to  call  in  during  class.  Data  were  collected  for  the  GED  cable 
class  during  eleven  90-minute  observation  periods  over  the  15-week  period. 

The  results  showed  that  the  teachers  did  adjust  their  teaching  styles  to 
make  better  use  of  the  technology. 

The  data  for  the  spring  and  fall  classes  for  the  math  and  language 
teachers  Illustrates  the  nature  of  the  changes  (see  Table  2).  In  the 
spring  the  mathematics  teacher  had  spent  almost  two-thirds  of  her  time 
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Table  2 

DISTRIBUTION  OF  ACTIVITIES  AND  INTERACTIONS  BY 
TEACHER  AND  CONDITION 


Mathematics  ) 

1 English  Language 

Distribution  of 

Classroom 

Cable 

Classroom 

Cable 

Activity  Time 

Spring 

Spring 

Fall 

Spring 

Spring 

Fall 

Teacher  Substantive 
Presentation 

16.4% 

18.5% 

24.3% 

0.0% 

1 

11.8%  1 

20.9% 

Classroom  Substantive 
Discussion 

22.7 

0.0 

0.0 

26.5 

0.0 

0.0 

Individual  Work 

1 

30.7 

65.9 

49.0 

32.3 

54.2 

42.3 

Drill  Activities 

13.0 

9.8 

17.5 

37.5 

27.1 

24.6 

Procedural  Information 

6.7 

5.8 

1 

6.5 

3.7 

6.9 

5.3 

Equipment  Adjustment 

0.0 

0.0 

1.6 

0.0 

1 

0.0 

0.6 

Teacher  Works  with 

Subgroup 

1.7 

0.0 

0.0 

0.0 

0.0 

0.0 

Nondeslgnated  Activity 

8.8 

0.0 

1.1 

0.0 

0.0 

6.3 

Total 

100.0% 

100.0% 

100.0% 

100.0% 

100.0% 

100.0% 

Frequency  of  Interaction 
per  Class 

Closed-ended  Questions 

2.1 

5.7 

32.5 

1 

32.4 

35.6 

1 

41.6 

Open-ended  Questions 

39.2 

0.0 

0.0 

50.3 

0.0 

1.4 

Questions  about 

Understanding 

34.1 

19.8 

9.8 

92.6 

3.1 

6.8 

Procedural  Questions 

14.5 

8.8 

24.0 

4.6 

21.9 

25.1 

Total  Frequencies 

89.9 

34.3 

66.3 

179.9 

60.6 

74.9 

Total  Minutes  of 

Observation 

211 

205 

423 

1 

136 

144 

333 
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with  Individual  work  periods  on  the  cable,  whereas  less  than  a third  of 
her  classroom  time  had  been  devoted  to  individual  work.  In  the  fall  she 
gave  half  the  cable  class  time  to  individual  work,  and  reallocated  the 
time  gained  to  drill.  That  shift,  coupled  with  increased  requests  that 
students  signal  their  answers  to  exercises,  doubled  the  average  number 
of  teacher-initiated  interactions.  The  math  teacher  asked  her  second 
cable  class  to  respond  to  closed-ended  questions  32.5  times  per  class, 
or  every  three  minutes. 

Examining  the  findings  for  the  language  class,  we  see  that  in  the 
spring  the  teacher  used  classroom  substantive  discussion  and  drill  for 
presenting  material  in  the  classroom.  This  was  also  captured  in  the  in- 
teraction record,  which  showed  that  she  was  rapidly  moving  through  an 
average  of  180  queries  per  class.  In  the  cable  class  she  had  used  relat- 
ively more  time  for  lecture  (11.8  percent)  and  individual  work  periods 
(54.2  percent  as  compared  with  32.3  percent)  as  well  as  devoting  less 
time  to  drill  over  the  cable.  In  the  spring  class  she  too  decreased  the 
amount  of  time  devoted  to  individual  study  by  being  more  self-conscious 
about  the  absence  of  cues  to  move  on.  The  time  saved  in  her  class  was 
Invested  in  lecture.  Increasing  that  activity  to  20.9  percent  of  her 
class  time. 

Taken  as  a whole,  these  findings  Indicated  that  classroom  observation 
data  can  be  useful  in  helping  teachers  become  aware  of  their  teaching 
styles.  In  addition,  the  data  indicate  that  teachers,  once  aware  of  the 
pedagogical  characteristics  of  their  class,  are  able  to  alter  their 
teaching  behaviors  to  better  reflect  their  own  notions  of  how  a class 
should  proceed. 

Seeking  Explanations  for  Student  Achievement 

The  results  of  the  second  class  again  found  that  students  attending 
the  electronic  classroom  made  satisfactory  gains.  In  looking  at  the 
classroom  dynamics  from  that  perspective,  several  explanations  begin  to 
emerge.  Although  there  is  a substantial  loss  of  class  discussion  and 
open-ended  questions  in  the  cable  class,  the  conventional  classroom  does 
not  spend  time  in  student  dominated  discussion,  rarely  does  the  teacher 
walk  around  the  class  to  work  with  individual  students,  and  the  students 
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initiate  very  few  questions.  The  observation  data  shows  that  the  teachers 
in  the  conventional  class  do  not  take  full  advantage  of  the  range  of 
educational  arrangements  possible  in  the  regular  classroom  environment,  but 
not  available  on  cable.  Thus,  the  absence  of  significant  differences  in 
educational  achievement  may  reflect  the  fact  that,  in  actuality,  the 
pedagogical  differences  between  the  two  conditions  may  not  have  been  as 
great  as  might  have  been  expected. 

Another  explanation  comes  from  considering  the  different  consequence 
of  a closed-ended  question  in  the  two  environments.  Although  these  data 
seem  to  indicate  that  students  in  the  cable  classroom  engage  in  much  less 
interactive  activity  than  those  in  the  conventional  class,  this  difference 
may  be  more  apparent  than  real.  When  a teacher  asks  a question  in  a con- 
ventional classroom,  only  one  or  two  students  need  to  respond.  The  other 
students  are  not  required  to  focus  on  the  question,  choose  an  answer,  and 
receive  reinforcement.  In  the  cable  classroom  with  home  data  terminals, 
every  student  in  the  class  answers  each  question  independently,  without 
guidance  or  cues  from  others.  Particularly  for  passive  students,  the 
nature  and  effect  of  this  structuring  of  student  attention  and  participa- 
tion may  be  a key  to  the  success  of  the  electronic  classroom. 

Results  of  the  Serial  Design 

When  a serial  design  is  used,  the  researcher  can  make  the  best  of 
several  worlds.  It  is  a prudent  strategy  which  can  allow  serious  re- 
structuring of  the  project  without  sacrificing  evaluation.  When  the 
concept  works  as  it  did  in  Spartanburg,  the  data  gives  one  confidence 
to  leave  well  enough  alone.  But  there  are  other  advantages. 

The  first  and  more  obvious  opportunity  is  that  one  can  pool  the 
data.  In  all,  we  offered  the  GED  course  in  Spartanburg  three  times.  We 
had  10,  12,  and  11  enroll  in  the  course,  and  fewer  took  the  ABLE  post- 
test. If  we  had  not  had  several  experiments,  kept  them  constant  in  terms 
of  teachers,  curriculum,  and  technology,  and  been  able  to  pool  the  data, 
the  conclusions  would  have  been  lame  and  tentative.  The  comparison  group 
of  students  taught  by  the  same  teachers,  using  the  same  workloads  and 
lectures,  adds  to  our  ability  to  reach  a confident  conclusion.  The  students 
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In  the  electronic  classroom  learned  as  much  as  the  students  in  the  con- 
ventional class.  Using  mean  percentile  ranks  on  the  ABLE  as  a criterion, 
we  see  on  Table  3 that  when  the  cable  class  began,  they  were  weaker  students. 
But  they  essentially  held  pace,  and  were  just  behind  the  conventional  class 
on  the  posttest.  Using  percent  gain  as  a criterion,  they  made  similar 
progress.  The  dropout  rate  was  slightly  higher  in  the  cable  class,  par- 
ticularly in  the  last  class  when  serious  technical  problems  plagued  the 
system.  Thus  there  is  the  possibility  that  more  weak  students  dropped 
from  the  cable  class,  artificially  enhancing  the  average  gains  based 
on  the  posttest.  Even  so,  that  is  not  likely  to  be  a strong  enough 
factor  to  alter  the  basic  conclusion:  the  proportionate  gains  of  the 
cable  were  not  statistically  different  from  the  conventional  classroom. 

The  conclusions  of  the  Spartanburg  project  support  efforts  to  re- 
plicate the  electronic  classroom  concept  for  home  education.  Since  the 
basic  communications  functions  of  the  system  are  outbound  audio-video 
and  return  data,  it  seems  clear  that  the  approach  could  use  broadcast 
television  and  telephone  return  and  other  technology  mixes  as  alternatives 
to  two-way  cable.  We  know  that  the  teachers  can,  with  limited  initial 
training,  adjust  their  teaching  styles  to  the  electronic  environment,  and 
something  about  the  problems  that  will  arise  when  teachers  are  cut  off  from 
visual  cues  about  their  class.  Such  classes  are  probably  equivalent  to 
classroom  instruction  as  it  normally  exists,  even  if  they  cannot  provide 
instruction  as  good  as  that  in  the  ideal  class. 

More  generally,  we  found  that  the  use  of  a serial  design  has  many 
advantages  and  would  strongly  recommend  it  for  future  field  work.  Serial 
experiments,  with  both  process  and  outcome  measures,  and  a standard  of 
comparison  served  the  combined  purposes  of  the  management  of  Innovation 
and  research  on  its  effects.  The  broad  conclusion  is  that  the  use  of 
serial  designs  and  a more  creative  approach  to  communications  system 
evaluation  will  increase  the  value  of  research  for  the  manager.  That, 
in  turn,  creates  incentives  for  more  controlled  management  of  the  project, 
enabling  the  evaluator  to  do  better  research. 


18 


Table  3 

STUDENT  ACHIEVEMENT  ON  ADULT  BASIC  LEARNING  EXAMINATION 
(mean  percentile  ranks) 


